Augmented reality z-stack prioritization

ABSTRACT

A system, method and computer program products for prioritizing the stacking order of virtualized objects within a z-index of an augmented reality system by presenting the most valuable, relevant or important information more prominently. Prioritization of the virtualized object may occur dynamically, as a function of the user&#39;s primary focus. As the user&#39;s primary focus changes from one physical object to another, the stacking order may change in response to the shift in focus. Embodiments may use one or more techniques to identify the primary focus such as the direction of the user&#39;s gaze, the focal point of one or more visual recording systems (i.e. cameras) or the user may manually highlight one or more objects via the HUD or interface of the augmented display system. The disclosed embodiments allow for the more predominate or relevant virtualized objects within the augmented display to receive priority, unobscured viewing by the user.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation application claiming priority to Ser.No. 15/834,125 filed Dec. 7, 2017, the contents of which are herebyincorporated by reference.

TECHNICAL FIELD

The present disclosure relates generally to augmented reality systems,methods and tools.

BACKGROUND

Augmented reality (AR) is a live direct or indirect view of a physicalreal-world environment whose elements of the physical world are modifiedwith computer generated sound, video or graphics when displayed througha computing system. AR is related to the concept of computer-mediatedreality, in which a view of reality is modified by a computer. Augmentedreality enhances one's current perception of reality, whereas incontrast, virtual reality replaces the real world with a simulated one.Augmentation techniques are typically performed in real time and insemantic context with environmental elements, such as overlayingsupplemental information over a live video feed.

With the help of advanced AR technology (e.g. adding computer vision andobject recognition) the information about the surrounding real world ofthe user becomes interactive and digitally manipulatable. Informationabout the environment and its objects are overlaid onto views and imagesof the real world. This information can be virtual or real. For example,seeing other real sensed or measured information such as electromagneticradio waves overlaid in exact alignment with where they are in space.

SUMMARY

A first embodiment of the present disclosure provides a method forprioritizing a stacking order of a z-index of virtual overlays of anaugmented reality system comprising the steps of: determining, by aprocessor, a primary focus of a user operating an augmented displaysystem; detecting, by a processor, one or more objects viewed throughthe augmented display system; designating, by the processor, each of theone or more objects as a primary object and a secondary object as afunction of the primary focus of the user; retrieving, by the processor,the virtual overlays from one or more data sources corresponding to eachprimary object and secondary object; assigning, by the processor, thestacking order for each of the virtual overlays in the z-index of aheads-up display (HUD) of the augmented display system generated by theprocessor, wherein a stacking order of the virtual overlays of theprimary object is higher than a stacking order of the virtual overlaysof each secondary object and; displaying, by the processor, each of thevirtual overlays in the z-index of the HUD in accordance with thestacking order assigned, and juxtaposing each of the virtual overlaysnear each corresponding primary object or secondary object.

A second embodiment of the present disclosure provides a computersystem, comprising: a processor; a memory device coupled to theprocessor; a visual recording system comprising a camera, coupled to theprocessor; a display device, coupled to the processor; and a computerreadable storage device coupled to the processor, wherein the storagedevice contains program code executable by the processor via the memorydevice to implement a method for prioritizing a stacking order of az-index of virtual overlays comprising the steps of: determining, by theprocessor, a primary focus of a user operating an augmented displaysystem; detecting, by a processor, one or more objects viewed throughthe augmented display system; designating, by the processor, each of theone or more objects as a primary object and a secondary object as afunction of the primary focus of the user; retrieving, by the processor,the virtual overlays from one or more data sources corresponding to eachprimary object and secondary object; assigning, by the processor, thestacking order for each of the virtual overlays in the z-index of aheads-up display (HUD) of the augmented display system generated by theprocessor, wherein a stacking order of the virtual overlays of theprimary object is higher than a stacking order of the virtual overlaysof each secondary object and; displaying, by the processor, each of thevirtual overlays in the z-index of the HUD in accordance with thestacking order assigned, and juxtaposing each of the virtual overlaysnear each corresponding primary object or secondary object.

A third embodiment of the present disclosure provides a computer programproduct comprising: one or more computer readable hardware storagedevices having computer readable program code stored therein, saidprogram code containing instructions executable by the one or morecentral processing units (CPU) of a computer system implementing amethod for comparing products using augmented reality comprising thesteps of: determining, by the CPU, a primary focus of a user operatingan augmented display system; detecting, by a CPU, one or more objectsviewed through the augmented display system; designating, by the CPU,each of the one or more objects as a primary object and a secondaryobject as a function of the primary focus of the user; retrieving, bythe CPU, the virtual overlays from one or more data sourcescorresponding to each primary object and secondary object; assigning, bythe CPU, the stacking order for each of the virtual overlays in thez-index of a heads-up display (HUD) of the augmented display systemgenerated by the CPU, wherein a stacking order of the virtual overlaysof the primary object is higher than a stacking order of the virtualoverlays of each secondary object and; displaying, by the CPU, each ofthe virtual overlays in the z-index of the HUD in accordance with thestacking order assigned, and juxtaposing each of the virtual overlaysnear each corresponding primary object or secondary object.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a block diagram of an embodiment of a system forprioritizing a stacking order of a z-index of virtual overlays ofaugmented reality.

FIG. 2a depicts an isometric view of an embodiment of an augmenteddisplay system.

FIG. 2b depicts a front view of the embodiment of the augmented displaysystem of FIG. 2 a.

FIG. 2c depicts a side view of the embodiment of the augmented displaysystem of FIG. 2 a.

FIG. 2d illustrates an embodiment of an augmented display systemequipped by a user 310.

FIG. 3a illustrates a front view of an alternative embodiment of anaugmented display system.

FIG. 3b illustrates a rear view of the alternative embodiment of theaugmented display system of FIG. 3 a.

FIG. 4a depicts a first person view of a user viewing an embodiment of aHUD of an augmented display system overlaying an embodiment of aplurality of virtual overlays having a prioritized z-index as a functionof the user's primary focus.

FIG. 4b depicts a first person view of a user viewing an embodiment of aHUD of an augmented display system overlaying an embodiment of aplurality of virtual overlays having an alternative prioritized z-indexas a function of the user's primary focus.

FIG. 4c depicts a first person view of a user viewing an embodiment of aHUD of an augmented display system overlaying an alternative embodimentof a plurality of virtual overlays have another alternative prioritizedz-index as a function of the user's primary focus.

FIG. 5 depicts a first person view of a user viewing a display device ofan alternative embodiment of an augmented display system overlaying aplurality of virtual overlays having a prioritized z-index onto avirtualized display of objects.

FIG. 6 illustrates a schematic view of an embodiment of a system forprioritizing a stacking order of a z-index of virtual overlays ofaugmented reality operating in a cloud computing environment.

FIG. 7 illustrates a schematic view of an embodiment of the abstractionlayers of a cloud computing environment.

FIG. 8 depicts an embodiment of an algorithm for prioritizing a stackingorder of a z-index of virtual overlays of augmented reality with thesystems, devices, methods and tools described throughout the presentdisclosure.

FIG. 9 depicts a block diagram of a computer system able to implementthe methods for prioritizing a stacking order of a z-index of virtualoverlays of augmented reality, consistent with the disclosure of thepresent application.

DETAILED DESCRIPTION

Although certain embodiments are shown and described in detail, itshould be understood that various changes and modifications may be madewithout departing from the scope of the appended claims. The scope ofthe present disclosure will in no way be limited to the number ofconstituting components, the materials thereof, the shapes thereof, therelative arrangement thereof, etc., and are disclosed simply as anexample of embodiments of the present disclosure. A more completeunderstanding of the present embodiments and advantages thereof may beacquired by referring to the following description taken in conjunctionwith the accompanying drawings, in which like reference numbers indicatelike features.

As a preface to the detailed description, it should be noted that, asused in this specification and the appended claims, the singular forms“a”, “an” and “the” include plural referents, unless the context clearlydictates otherwise.

Overview

Augmented reality (AR) is continuing to become increasingly popular asAR technologies improve and advance. AR systems such as Google glass,Samsung Gear and a—plethora of associated AR applications have becomemore prolific in view of the advancements in AR and computing technologygenerally. However, currently available AR hardware, software andplatforms are still imperfect. One noticeable flaw in the currentlyavailable AR technology may be due to the insufficient ability of ARsystems and software to dynamically prioritize the stacking order ofz-indexes providing virtual overlays of information, graphics, images orother computerized objected applied to the augmented HUD. The lack ofprioritization in the stacking order may cause augmented reality objectsthat overlay the augmented world to obstruct the view of important andvaluable information that is being presented by the AR systems.Therefore, there is a need for prioritizing the stacking order of thevirtual overlays providing virtual objects within the augmented realityenvironment based on the importance of the information found within thevirtual overlay to the user. Thus, improving the current AR systems,hardware and software by preventing more important information overlaidor juxtaposed in the augmented reality system's display device frombeing obstructed due to an implementation of a chronological, archaic,or random prioritization methods.

Embodiments of the present disclosure understand the need for improvingaugmented reality. Embodiments of the present disclosure improvecurrently available augmented reality systems, methods and tools byprioritizing the stacking order of virtualized objects (referred to asvirtual overlays) within a z-index in an order that presents the mostvaluable, relevant or important information more prominently and thushigher in the stacking order. A stacking order may refer to arelationship between one or more virtualized objects and the order inwhich certain layers of the virtualized objects may obscure the view ofothers within the same virtual space of augmented reality viewed througha HUD. For example, two virtual objects, such as windows of an operatingsystem or graphical user interface (GUI) may have a set stacking order,wherein the currently targeted window is positioned on the top (highestpriority) over other windows that may be currently open. In the currentdisclosure, the virtualized objects are one or more different objectsthat may contain graphics, images, video, text, etc., creating thevirtual overlays that may be presented on a HUD of a display device 110using an augmented display system 103.

The embodiments of the present disclosure may prioritize the stackingorder of one or more virtual overlays dynamically, as a function of theuser's 301 primary focus 401, while assigning secondary objects 406 a,406 b, 406 c (referred to generally as “secondary objects 406”) ortertiary objects 405 a, 405 b, 405 c, 405 d, 405 e (referred togenerally as “tertiary objects 405”) to a lower level of the z-index. Asthe user's 301 primary focus 401 changes from one primary object 403 toanother, the stacking order of the z-index may also change. The primaryfocus 401 on the primary object 403 may have the highest priority in thestacking order of the z-index. The primary object 403 may be determinedto be (at least initially) the most important object to the user 301within the current viewing distance of the augmented display system 103.Embodiments of the present disclosure may use one or more techniques toidentify the primary focus 401 of the user 301. For instance, theaugmented reality systems 103 may estimate the user's 301 primary focus401 based on the direction of the user's 301 gaze, the focal point ofone or more visual recording systems 124 (i.e. cameras) integrated intoan augmented display system 103, or the user may manually highlight oneor more objects via the HUD or interface of the augmented display system103.

The next layer below the primary objects 403 that are prioritized withinthe stacking order of the z-index may comprises secondary objects 406and the secondary overlays 409 corresponding to each secondary object406. Secondary objects 406 may be one or more objects within the viewingdistance or range of an augmented display system 103 which may be taggedor identified as being related to a primary object 403 in some manner.For example, the secondary object 406 may be a similar type of productoffered by a competitor, another product offering under the same brandas the primary object 403 or an object tagged with similar keywords ormetadata as a primary object 403. In some instances, multiple secondaryobjects 406 may be present. Embodiments of the present disclosure mayprioritize the stacking order of multiple secondary objects 406 and eachcorresponding secondary overlays 409 based one or more secondaryfactors. Embodiments of the secondary factors may include (but are notlimited to) a value of a particular promotion (i.e. best promotionaldeal may be prioritized), relevance to the user's 301 browsing historyand preferences, social relationships and recommendations, purchasehistory, the user's surrounding environment, and the cognitive state ofthe user 301.

The next layer below the secondary objects in the stacking order may betertiary objects 405 and each tertiary overlay 411 a , 411 b (referredgenerally as tertiary overlay 411) corresponding to each tertiary object405. Embodiments of tertiary objects 405 may include one or more objectsdetected within the viewing distance of a user operating an augmenteddisplay system 103. However, unlike the secondary objects 406 which maybear some related ancillary or related characteristic to the primaryobject 403 of the user's 301 primary focus 401, a tertiary object maynot. For instance, when shopping at a store, a store shelf in theelectronics department may have numerous different types of gadgets andelectrical devices within the viewing distance of the augmented displaysystem 103. While the user may find some of the items interesting anduseful, the tertiary objects 405 may not currently be closely relatedenough to the primary object 403 to warrant a higher priority in thez-index. Therefore, while tertiary objects 405 and each correspondingtertiary overlay 411 may be displayed within the HUD of the displaydevice 110 of the augmented display system 103, tertiary objects 405 mayhave the lowest priority and be obscured from view by primary objects403, primary overlays 407, secondary objects 406 and secondary overlays409.

However, similar to the secondary objects 406, a plurality of tertiaryobjects 405 may exist within the z-index. In some embodiments, theaugmented display system 103 may further prioritize one or more tertiaryobjects 405 and each corresponding tertiary overlay 411, relative to theremaining tertiary objects 405 and corresponding tertiary overlays 411.Prioritization among the co-equal tertiary objects 405 may occur in amanner similar to the prioritization amongst the secondary objects 406described above. For example, each tertiary object 405 and the substanceof the corresponding tertiary overlays 411 may be compared for secondaryfactors that may be identified as making one tertiary object 405 moreimportant to the user 301 than another tertiary object 405. Thesesecondary factors to consider by the augmented display system 103 mayinclude the value of a promotion tied to an object (i.e. best or mostvalue provided by a promotional deal may be prioritized and thus bepresented higher in a stacking order), the relevance of the tertiaryobject 405 to the user's 301 browsing history or preferences, mentionsof the tertiary object by individuals with a social relationship to theuser 301 (i.e. via social media), recommendations by others, purchasehistory of the user 301, the user's 301 surrounding environment, and thecognitive state of the user 301.

Using the primary focus 401 of the user 301 as a starting point, anaugmented display system 103 may detect one or more objects ordepictions of objects (such as signs, billboards, images, etc.) withinthe viewing distance user's primary focus 401. The augmented displaysystem 103 may categorize each object as a primary object, 403,secondary object 406 or tertiary object 405. Embodiments of theaugmented display system 103 may retrieve or load one or more virtualoverlays into a memory device 114 of the augmented display system 114and generate a GUI of a HUD comprising each of the virtual overlays.Embodiments of the augmented display system 103 may assign a stackingorder within the z-index for each object detected along with thecorresponding virtual overlay that is juxtaposed onto the GUI of theHUD. The augmented display system 103 may prioritize the primary object403 having a primary overlay 407 as the highest layer of the stackingorder. Subsequently, the next highest layer may include the secondaryobjects 406 and each corresponding secondary overlays 409, wherein anyoverlap between the primary overlay 407 and the secondary overlay 409may result in the primary overlay 407 obscuring all or a portion of thesecondary overlay 407 due to the prioritization of the primary overlayin the stacking order of the z-index. Lastly, each tertiary object 405and corresponding tertiary overlay 411 may also be juxtaposed onto theGUI of the HUD. Any portions of the tertiary overlay overlapping withinthe same virtual space as either a primary overlay 407 or a secondaryoverlay 409 may be partially or entirely obscured from the user's view,allowing for the more predominate primary overlay 407 or secondaryoverlay to receiving priority viewing by the user 301.

System for Prioritizing Stacking Order of a Z-Index

Referring to the drawings, FIG. 1 illustrates a diagram of an embodimentof a system 100 for prioritizing a stacking order of a z-index ofvirtual overlays using augmented reality, consistent with thedisclosures of this application. Embodiments of system 100 may comprisespecialized computer systems referred to as augmented display systems103, which may each have a specialized configuration of hardware,software or a combination thereof as depicted in FIGS. 1-7 and asdescribed throughout the present disclosure. Embodiments of theaugmented display systems 103 may each comprise one or more elements ofa generic computer system 900 of FIG. 9 (described in detail below). Oneor more of the elements of the generic computer system 900 may beintegrated into each of the augmented display systems 103 describedherein or other computer systems of system 100, for example networkrepository 133 or a computer system maintaining an overlay data source131 a, 131 . . . 131 n (referred to generally as overlay data source131) such as a server or other network accessible computer system.

Embodiments of the augmented display systems 103 may be a specializedcomputer system which may include one or more processors 116,specialized hardware or circuitry and/or software loaded in the memorydevice 114. The embodiments of the augmented display system 103 mayperform functions, tasks and routines relating to the detection andviewing of each object or representations of objects, querying overlaydata sources 131 to retrieved one or more virtual overlays relating tothe objects being viewed by the augmented display system 103, retrievingsaid virtual overlays from the overlay data sources 131 or networkrepository 133, assigning categories to each object being viewed,generating a GUI of a HUD comprising the virtual overlays juxtaposingeach object corresponding object, prioritizing the stack order of eachvirtual overlay in the z-stack of HUD's GUI and displaying the augmentedreality within the HUD comprising the physical, virtual orrepresentations of the viewed objects and each virtual overlay as afunction of the z-index's stacking order.

Embodiments of the augmented display systems 103 may be connected andplaced in communication with one or more additional computer systems orhardware, over a computer network 120. Embodiments of the network 120may be constructed using wired or wireless connections between eachhardware component or virtualized hardware connected to the network 120.As shown in the exemplary embodiment of FIG. 1, each of the augmenteddisplay systems 103 may connect to the network 120 and communicate overthe network 120 with additional computer systems such as overlay datasources 131 and network repositories 133 using a network interfacecontroller (NIC) 122 or another network communication device.

Embodiments of the NIC 122 may implement specialized electroniccircuitry allowing for communication between each member of the network120 using a specific physical layer and a data link layer standard, suchas Ethernet, Fiber channel, Wi-Fi or Token Ring. The NIC 122 may furtherallow for a full network protocol stack, enabling communication overnetwork 120 to the augmented display systems 103, overlay data sources131 (such as servers hosting virtual overlays and other graphicalobjects), network repositories 133 and other computing hardware deviceslinked together through communication channels. The network 120 mayfacilitate communication and resource sharing among the augmenteddisplay system 103, overlay data sources 131 and additional hardwaredevices connected to the network 120, for example a network repository133 or other network accessible storage devices connected to the network120. Examples of network 120 may include a local area network (LAN),home area network (HAN), wide area network (WAN), back bone networks(BBN), peer to peer networks (P2P), campus networks, enterprisenetworks, the Internet, cloud computing networks and any other networkknown by a person skilled in the art.

In some embodiments, the network 120 may be a cloud computingenvironment 50. Cloud computing is a model of service delivery enablingconvenient, on-demand network access to a shared pool of configurablecomputing resources (e.g., networks, network bandwidth, servers,processing, memory, storage, applications, virtual machines, andservices) that can be rapidly provisioned and released with minimalmanagement effort or interaction with a provider of the service. Thiscloud model may include at least five characteristics, at least threeservice models, and at least four deployment models. The characteristicsof the cloud computing model may be described as follows:

On-demand self-service: a cloud consumer can unilaterally provisioncomputing capabilities, such as server time and network storage, asneeded automatically without requiring human interaction with theservice's provider.

Broad network access: capabilities are available over a network andaccessed through standard mechanisms that promote use by heterogeneousthin or thick client platforms.

Resource pooling: the provider's computing resources are pooled to servemultiple consumers using a multi-tenant model, with different physicaland virtual resources dynamically assigned and reassigned according todemand. There is a sense of location independence in that the consumergenerally has no control or knowledge over the exact location of theprovided resources but may be able to specify location at a higher levelof abstraction (e.g., country, state, or datacenter).

Rapid elasticity: capabilities can be rapidly and elasticallyprovisioned, in some cases automatically, to quickly scale out andrapidly released to quickly scale in. To the consumer, the capabilitiesavailable for provisioning often appear to be unlimited and can bepurchased in any quantity at any time.

Measured service: cloud systems automatically control and optimizeresource use by leveraging a metering capability at some level ofabstraction appropriate to the type of service (e.g., storage,processing, bandwidth, and active user accounts). Resource usage can bemonitored, controlled, and reported, providing transparency for both theprovider and consumer of the utilized service.

The service models under a cloud computing environment 50 may bedescribed as follows:

Software as a Service (SaaS): the capability provided to the consumer isto use the provider's applications running on a cloud infrastructure.The applications are accessible from augmented display devices 103through a thin client interface such as a web browser (e.g., web-basede-mail). The consumer does not manage or control the underlying cloudinfrastructure including network, servers, operating systems, storage,or even individual application capabilities, with the possible exceptionof limited user-specific application configuration settings.

Platform as a Service (PaaS): the capability provided to the consumer isto deploy onto the cloud infrastructure consumer-created or acquiredapplications created using programming languages and tools supported bythe provider. The consumer does not manage or control the underlyingcloud infrastructure including networks, servers, operating systems, orstorage, but has control over the deployed applications and possiblyapplication hosting environment configurations.

Infrastructure as a Service (IaaS): the capability provided to theconsumer is to provision processing, storage, networks, and otherfundamental computing resources where the consumer is able to deploy andrun arbitrary software, which can include operating systems andapplications. The consumer does not manage or control the underlyingcloud infrastructure but has control over operating systems, storage,deployed applications, and possibly limited control of select networkingcomponents (e.g., host firewalls).

The deployment models of cloud computing environments may be describedas follows:

Private cloud: the cloud infrastructure is operated solely for anorganization. It may be managed by the organization or a third party andmay exist on-premises or off-premises.

Community cloud: the cloud infrastructure is shared by severalorganizations and supports a specific community that has shared concerns(e.g., mission, security requirements, policy, and complianceconsiderations). It may be managed by the organizations or a third partyand may exist on-premises or off-premises.

Public cloud: the cloud infrastructure is made available to the generalpublic or a large industry group and is owned by an organization sellingcloud services.

Hybrid cloud: the cloud infrastructure is a composition of two or moreclouds (private, community, or public) that remain unique entities butare bound together by standardized or proprietary technology thatenables data and application portability (e.g., cloud bursting forload-balancing between clouds).

A cloud computing environment 50 may be service oriented with a focus onstatelessness, low coupling, modularity, and semantic interoperability.At the heart of cloud computing is an infrastructure that includes anetwork of interconnected nodes 10. Referring to the drawings, FIG. 6 isillustrative of a network 120 operating as a cloud computing environment50. As shown, the cloud computing environment 50 may include one or morecloud computing nodes 10 with which client computing devices, such asone or more augmented display systems 103 a, 103 b, 103 c. . . 103 n(referred to collectively as “augmented display systems 103”) used bycloud consumers, such as, for example, desktop computers 103 c, laptopcomputers 103 a, and mobile communication devices 103 b, tabletcomputers or computer systems integrated into devices as exemplified bythe smart glasses 103 n shown in FIG. 6.

Computer system nodes 10 of the cloud computing environment 50 maycommunicate with one another and may be grouped physically or virtually(not shown), in one or more networks, such as Private, Community,Public, or Hybrid clouds as described hereinabove, or a combinationthereof, allowing for the cloud computing environment 50 to offerinfrastructure, platforms and/or software as services for which a cloudconsumer does not need to maintain resources on an augmented displaydevice 103. It is understood that the types of augmented display devices103 as shown in FIGS. 1-5 are intended to be illustrative only and thatnodes 10 of a cloud computing environment 50 can communicate with anytype of augmented display devices 103 over any type of network 120and/or network addressable connection (e.g., using a web browser).

Referring now to FIG. 7, a set of functional abstraction layers providedby a cloud computing environment 50 of the network 120 is shown. Itshould be understood in advance that the components, layers, andfunctions shown in FIG. 7 are intended to be illustrative only andembodiments of the invention are not limited thereto. As depicted, thefollowing layers and corresponding functions are provided:

Hardware and software layer 60 includes hardware and softwarecomponents. Examples of hardware components include: mainframes 61; RISC(Reduced Instruction Set Computer) architecture based servers 62;servers 63; blade servers 64; storage devices 65; and networkingcomponents 66. In some embodiments, software components may includenetwork application server software 67 and database software 68.

Virtualization layer 70 provides an abstraction layer from which thefollowing examples of virtual entities may be provided: virtual servers71; virtual storage 72; virtual networks 73, including virtual privatenetworks; virtual applications and operating systems 74; and virtualclients 75.

Embodiments of the management layer 80 may provide the functionsdescribed below. Resource provisioning 81 provides dynamic procurementof computing resources and other resources that are utilized to performtasks within the cloud computing environment. Metering and Pricing 82provide cost tracking as resources are utilized within the cloudcomputing environment, and billing or invoicing for consumption of theseresources. In one example, these resources may include applicationsoftware licenses. Security provides identity verification for cloudconsumers and tasks, as well as protection for data and other resources.User portal 83 provides access to the cloud computing environment of thenetwork 120 for consumers (i.e. users 301) and system administrators.Service level management 84 provides cloud computing resource allocationand management such that required service levels are met. Service LevelAgreement (SLA) planning and fulfillment 85 provides pre-arrangementfor, and procurement of, cloud computing resources for which a futurerequirement is anticipated in accordance with an SLA.

Workloads layer 90 provides examples of functionality for which thecloud computing environment may be utilized. Examples of workloads andfunctions which may be provided from this layer include: recordsmanagement 91; web page management 92; searching and results management93; data analytics processing 94; profile management 95; and ARprioritization module 105.

Embodiments of the system 100, and augmented reality functionsprioritizing the stacking order of one or more virtual overlays viewedusing system 100, may be managed and controlled by an augmented displaysystem 103. The augmented display system 103 may be responsible fordetecting the presence of each object viewed within the viewing range ofthe augmented display system 103, detecting the primary focus 401 of theuser 301, identifying any secondary objects 406 or tertiary objects 405within range of the primary objects 403 that are the subject of theuser's primary focus 401, querying and retrieving virtual overlays 407,409, 411 corresponding to each primary object 403, secondary object 406and each tertiary object 405. The augmented display system 103 maygenerate and display a HUD that may be viewable by the user, wherein theHUD may comprise a mixture of graphics, images, video and text of thevirtual overlays 407, 409, 411 juxtaposed onto the one or more objects403, 405, 406 or representations thereof (real or virtual). During thegeneration and display of the HUD, the augmented display system 103 maybe responsible for creating or adjusting the stacking order of eachvirtual overlays 407, 409, 411 wherein the primary overlays 407 have thehighest priority, the secondary overlays 409 have the next priority,followed by the tertiary overlays 411 which may have the lowestpriority.

Embodiments of the augmented display system 103 may include specializedhardware and/or software integrated into the augmented display system103 performing each of the functions of the augmented display system 103mentioned above and throughout this application. The specializedcomponents of the augmented display system 103, implementing eachfunction or task may be part of an AR prioritization module 105. Thehardware and/or software components of the AR prioritization module 105may include one or more sub modules in some embodiments. These submodules may include a profile module 107, visual recognition module 108,display controller 109, overlay module 113, HUD module 115, speechrecognition module 117 and analytics module 119. As used herein, theterm “module” may refer to a hardware module, software-based module or amodule may be a combination of hardware and software resources of acomputer system and/or resources remotely accessible to the augmenteddisplay system 103 via the computer network 120.

Embodiments of the modules or sub modules described in this application,whether comprising hardware, software or a combination of resourcesthereof, may be designed to implement or execute one or more particularfunctions, tasks or routines of the augmented display system 103described herein. Embodiments of hardware-based modules may includeself-contained components such as chipsets, specialized circuitry andone or more memory devices comprising a memory storage medium (describedbelow). A software-based module may be part of a program code or linkedto program code or computer code 997, 998 containing specific programmedinstructions loaded into a memory device 114 of the augmented displaysystem 103, and/or a remotely accessible memory device 114 of anothernetwork accessible computer system accessed remotely by the augmenteddisplay system 103 over the network 120. For example, in someembodiments the network accessible computer system connected to theaugmented display system 103 may be a web server, application server,remote client device or other network accessible hardware.

Embodiments of the augmented display system 103 may include an ARprioritization module 105 which may be responsible for implementing oneor more functions or tasks of the augmented display system 103. Someembodiments of the AR prioritization module 105 of the augmented displaysystem 103 may include a profile module 107. The profile module 107 maybe responsible for performing the tasks and functions of managing eachindividual user profile corresponding to the current user 301 operatingthe augmented display device 103. A user 310 of the augmented displaysystem may identify a particular user profile corresponding the user301's viewing habits, searches, purchase history, history of viewedproducts as well as social media interactions, email or messagingdiscussions. The profile module 107 may load the user profile 130corresponding to the current user 301 of the augmented display device103 into the memory device 114 of the augmented display system 103. Uponloading the appropriate user profile 130, custom configurations,settings, product search histories and past user experiences, includinga log file describing the types of objects previously focused on by theuser, may be accessible. The use of separate and distinct user profiles130 may allow for multiple users 301 of a single augmented displaysystem 103 to separately store and recall their own distinct userinformation and product viewing histories, thus providing a customizedexperience, more accurate or relevant object prioritization unique toeach user's own habits of use.

In some embodiments, the profile module 107 may retrieve user profilesstored remotely on the network 120, for example in a network repository133. By storing user profiles 130 remotely, a user 301 may not be tiedto a specific augmented display system 103. Rather, even if a user 301changes the augmented display device the user 301 is using, the profilemodule 107 may retrieve and update the same user profile 130 across allaugmented display systems 103 used by a specific user 301. Thus,allowing for a user 301 to load customized settings and build acontinuous and seamless viewing and purchase history even whenperiodically using different augmented display systems 103.

Embodiments of the AR prioritization module 105 may comprise a visualrecognition module 108. The visual recognition module 108 may beresponsible for controlling the visual recording system 124,establishing a primary focus 401 of the user 301, detecting each type ofobject 403, 405, 406 viewed using the visual recording system 124. Thevisual recording system 124 may include camera systems, video processinghardware/software and video recording hardware/software. Embodiments ofthe visual recording system 124 may record or stream visual images andvideo data from an input device of the visual recording system 124 (suchas a camera) to the visual recognition module 108. The visualrecognition module 108 may process the recorded or streamed visualimages and/or video data to identify the primary focus 401 of the user,a primary object 403 that may be associated with the user's primaryfocus 401, secondary objects 406 that may be categorically, functionallyor commercially related to the primary object 403, such as being thesame brand, a competing product, identified with similar keywords,similarly tagged or comprising identifiers that may be similar to theprimary object 403 and within the frame of the visual images or videodata being recorded or streamed. The visual recognition module 108 mayalso identify and any tertiary objects 405 that may be unrelated to theprimary object 403, but may still be within the frames of the visualimages or video data captured by the visual recording system 124.

Embodiments of the visual recognition module 108 may identify theprimary focus 401 of a user 301 and each type of object 403, 407, 409within the viewing range of the augmented display system 103 using oneor more techniques for detecting the focus of a user 301. For example,the visual recognition module 108 may calculate the positioning of theuser's eye gaze, use image recognition to identify the closest objectsto the user 310, location based solutions which may identify the productof interest based on the known location of the user 301 or by allowingthe user 301 of the augmented display system 103 to manually select anoption which indicates a user's 301 primary focus 401 on a specificprimary object 403 being viewed through the camera of the visualrecording system 124.

In some embodiments, the visual recognition module 108 may includeobject recognition software that may implement algorithms and mechanismsthat allow for the augmented display system 103 to identify each of theobjects 403, 407, 409 being viewed by the visual recording system 124.The object recognition element of the visual recognition module 108 mayutilize matching, learning, or pattern recognition algorithms usingappearance-based or feature-based techniques to identify not only theprimary object 403 being focused on by the user of the visual recordingsystem 124, but also surrounding objects that may be related to theprimary object 403 (i.e. secondary objects 406) as well as one or moreun-related tertiary objects that may be littered around the primaryobject 403. Common techniques implementing object recognition mayinclude deep learning based approaches such as convolutional neuralnetworks (CNN), feature-based approaches using edges, gradients,histogram of oriented gradients (HOG), Haar wavelets, and linear binarypatterns. Some object recognition techniques may implement a variety oflearning models including feature extraction, machine learning models,deep learning models, Bag-of-words models such as SURF and MSER,derivative based matching approaches, the Viola-Jones algorithm,template matching, image segmentation and/or blob analysis.

Embodiments of the visual recognition module 108 may draw conclusionsdirected toward which objects 403, 405, 406 are the focus of the user301 of the augmented display system 103 as well as the identification ofthe objects in order to further obtain additional information and visualdata to overlay or juxtapose onto a HUD of the augmented displaysystem's 103 display device 110. Embodiments of the visual recognitionmodule 108 may integrate analytics engine or separate analytics module119 to analyze and draw conclusions about the primary focus 401 of theuser 301, the identification of one or more objects 403, 405, 406 andwhether or not each of the objects 403, 405, 406 are somehow related tothe primary object 403 that is the primary focus 401 of the user 301.Embodiments of the visual recognition module 108, may, based on theidentification of each object 403, 405, 406, categorize or tag theidentified objects 401 with keywords, meta tags or identifyingindicators which may be used later on in order to classify the objectsas being a secondary object 406 that may be related to a primary objectthat has been previously identified and/or tagged with the same orrelated key words, tags or indicators. Based on the identification ofthe objects 403, 405, 406 within the viewing distance of augmenteddisplay system 103, the visual recognition module 108 may call theoverlay module 113 (discussed below) to retrieve additional informationabout the identified objects for display to the user 301 via the HUD ofthe display device 110.

In some embodiments of the augmented display system 103, the ARprioritization module 105 may comprise a display controller 109. Adisplay controller 109 may be an electrical circuit that may perform thetask or function of actuating the display device 110 and deliver thestream of images or video data recorded by the visual recording system124 to the display device 110. Upon initiating the visual recordingsystem 124, the image and video data may be transmitted from the visualrecording system's camera(s) to the visual recognition module 108 and/orthe display controller 109. The display controller 109 may deliver theframes content of the images and video being recorded by the visualrecording system 124 to the display device 110, allowing for the user301 of the augmented display system 103 to view the images and/or videodata as the data is being inputted into the camera system of the visualrecording system 124. Thus, allowing for a real-time viewing of theimage and video data of each object 403, 405, 406 being viewed in reallife or as visual representations 501 on the display device 110 of theaugmented display system 103.

Embodiments of the AR prioritization module 105 may, in some embodimentsmay comprise an overlay module 113. The overlay module 113 may performthe task or function of retrieving information, graphics, text or otherdata (described as a virtual overlay 407, 409, 411) corresponding toeach object 403, 405, 406 identified by the visual recognition module108. The graphics, text and information that comprise each virtualoverlay 407, 409, 411 may be generated and juxtaposed onto the HUD ofthe display device's 110 GUI using the virtual overlay data retrievedfrom or more locations. Locations that may store virtual overlay datamay include one or more overlay data sources 131, a local data storagedevices 118 and/or network repositories 133. Embodiments of the overlaymodule 113 may query the system 100 for data that comprises the virtualoverlays 407, 409, 411. Examples of overlay data sources 131 that may bequeried by the overlay module 113 in order to retrieve the virtualoverlay 407, 409, 411 data may include, but are not limited to searchinga manufacturers website, accessible product servers, online retailstores, local and network accessible database records, third partywebsites, search engines, internet forums, and social media. Upon asuccessful query, the overlay module 113 may download the virtualoverlays 407, 409, 411 to the augmented display system 103 or networkaccessible storage device such as network repository 133. The virtualoverlay 407, 409, 411 may be transmitted to the HUD module 115(described below) for augmented display onto the display device 110overlaying or juxtaposing the virtual overlays 407, 409, 411 in therelative vicinity of the collected set of images and video data beingdisplayed from the visual recording system 124.

The virtual overlays 407, 409, 411 data retrieved by the overlay module113 may also be cataloged, registered and stored in a database or otherdata structure for quick retrieval of the virtual overlays 407, 409, 411at a later point in time, if needed. The entry of each virtual overlay407, 409, 411 may be tagged with keywords, Meta tags, identifiers and abrief description of the corresponding object, allowing for quicksearches and retrievals of a specific virtual overlay, if an objectbecomes a primary object 403, secondary object 406 or tertiary object405 at a subsequent point in time.

The overlay module 113 may in some instances receive a system call fromthe visual recognition module 108 requesting the overlay module 113 tosearch a database of virtual overlays 407, 409, 411 for one or morevirtual overlays 407, 409, 411 associated with objects 403, 405, 406currently within the viewing distance of the augmented display system103. The overlay module 113 may query the database of stored virtualoverlays 407, 409, 411 by descriptor, keyword, category and meta tags,and transmit the virtual overlays 407, 409, 411 to the HUD module 115for augmented display 103 for prioritization of the stacking order anddisplay on a HUD of a display device 110.

Embodiments of the augmented display system 103 may further comprise aheads-up display module 115 (abbreviated as “HUD Module 115”). The HUDmodule 115 may perform the task or function of generating a GUI of a HUDoverlaying and/or juxtaposing one or more virtual overlays 407, 409, 411onto the display device 110 within the vicinity of the images and/orvideo data provided by the visual recording system 124. The HUD module115 may also prioritize the stacking order of each GUI element withinthe HUD including each virtual overlay 407, 409, 411 and data providedby the visual recording system 124. An example of the HUD module 115overlaying or juxtaposing virtual overlays 407, 409, 411 onto a HUD ofthe augmented display system's 103 GUI can be seen in FIGS. 4a -5. Eachof the virtual overlays 407, 409, 411 may be juxtaposed in a positioncorresponding to one or more images and video data of objects 403, 405,406 (either real or depictions of object such as in print media orartistic renderings) or digital representations 503, 505, 506 ofobjects.

Referring to the drawings, FIG. 4a depicts an example of an embodimentof a HUD being overlaid or juxtaposed near one or more objects 403, 405,406 within a display device 110 by generating a GUI capable ofdynamically displaying one or more virtual overlays 407, 409, 411corresponding to one or more primary 403, secondary 406 or tertiary 405objects in a prioritized stacking order as a function of a the user's301 primary focus 401 on a primary object 403. As shown in FIG. 4a , auser 301 may focus on a primary object 403 which is currently within theviewing range of the augmented display system 103. The primary focus 401of the user 301 operating the augmented display system 103 is depictedby the dashed line surrounding the primary object 403. An object whichmay have been identified by the visual recognition module 108. The HUDmodule 115 may position a primary overlay 407 containing informationabout the primary object 403 (in this case a computer monitor) at thehighest level of the stacking order within the z-index. In this example,the user 301 may be shopping for a new monitor and the system 100 may beprioritizing the virtual overlays 407, 409 of each available monitor inthe stacking order so that each monitor is unobstructed by other nearbyproducts that may be unrelated but might still be of some ancillaryinterest to the user 301.

The prioritized levels of stacking order can be seen by the fact thatthe primary overlay 407 partially obscures the tertiary overlay 411 acorresponding to the tertiary object 405 e which is lower in thestacking order due to not being the primary focus 401. Moreover, theexample also demonstrates that a HUD module 115 may also place one ormore secondary overlays 409 within the HUD appearing on the displaydevice 110, which may contain information or data associated with thesecondary object 406 a as shown. The secondary object 406 a may beconsidered higher in the stacking order and thus partially obscure theview of other nearby tertiary objects 405 b, 405 c or tertiary overlays411 in favor of prioritizing the display of the secondary overlay 409within the HUD. In this example, object 406 a may be considered asecondary object 406 a and thus receive higher prioritization becausesecondary object 406 a is another monitor and thus categorically relatedin object type to the primary object 403 which is also a monitor,whereas tertiary objects 405 a, 405 b and 405 c are mobile computingsystems.

The HUD module's prioritization of one or more virtual overlays 407,409, 411 may dynamically change as the user 301 changes the primaryfocus 401 to different objects within the viewing distance of theaugmented display system 103. FIG. 4b demonstrates an example of one ormore changes that may occur within the stacking order of the z-index asa result of the change in primary focus 401 from a first object (amonitor) to a second object, a tablet computer (i.e. an Apple® iPad) inthis second example. As seen by the difference between the shift inprimary focus 401 of the user 301 from the monitor in FIG. 4a to theiPad in FIG. 4b , the iPad, when the iPad become the primary focus 401the HUD module may dynamically reprioritize the stacking order of eachvirtual overlay 407, 409, 411 and reassign objects to a differentpriority. In the case of FIGS. 4a to 4b , the iPad changes from being atertiary object 405 e in FIG. 4a to a primary object 403 in FIG. 4b andthus has the highest priority in the stacking order. The change inpriority can be seen by the difference between the virtual overlayscorresponding to the objects. As the iPad becomes the primary focus 401,the corresponding virtual overlay becomes a primary overlay 407 and nowobscures the view of tertiary overlay 411 a associated with the monitorwhich has become a tertiary object 405 b in FIG. 4 b

Moreover, in FIG. 4b , the previously obscured mobile computing systemsin FIG. 4b may be classified as secondary objects 406 in the revisedprioritization of FIG. 4b . The mobile computing systems may beconsidered categorically or similarly enough related to the tabletcomputer of the iPad to be assigned a higher priority in the stackingorder over one or more of the monitors which have been subsequentlyassigned as tertiary objects in FIG. 4b . As a consequence of the changein stacking order, the monitor overlay previously associated with theView Sonic monitor in FIG. 4a become obscured and less visible withinFIG. 4b as it has been deemed by the HUD module 115 to be less importantor relevant as the mobile computing devices.

Referring now to FIG. 4c , in some embodiments, the HUD module 115 maybe responsible for also prioritizing the stacking order of co-equalobjects within the same assigned level of object prioritization. Forexample, as shown in FIG. 4c , the primary focus 401 of the user 301 hasshifted in FIG. 4c from the iPad (in FIG. 4b ) to one of the mobilecomputing devices. As a result of the shift in the user's 301 primaryfocus 401, there are now multiple mobile computing devices that may beconsidered secondary objects 406 a, 406 b because they are categoricallyrelated to the mobile computing device of the primary object 403 andthus vying for priority within the stacking order of the z-index.Accordingly, the HUD module 115 may prioritize one or more objectswithin the same object classification (primary, secondary, tertiary,etc.) using one or more secondary factors. Examples of secondary factorsthat may be used to prioritize co-equal objects may include, but is notlimited to the value of a promotional offer, relevance of the object tothe user (i.e. preferred brands or previously bought brands), socialrelationships (i.e. the user's 301 friends have talked about one objectover another), recommendations or reviews the user 301 may have read,user purchase history, cognitive state of the user (i.e. the system 100may prioritize objects that may make the user 301 feel better,energized, cope, etc.), the surrounding environment (i.e. is it sunny,rainy, hot, cold, snowy, etc., and thus prioritize objects that may bemore responsive or relevant to the current environment, for exampleclothing and accessories suitable for the current weather outside may bestacked higher in the stacking order when viewed through the augmenteddisplay system 103).

In the example of FIG. 4c , the primary focus 401 of the user 301 may bethe Blackberry® mobile computing device. The prioritization of theco-equal secondary objects 406 a, 406 b, in this case an Apple iPhone®and the Samsung Galaxy®, may be prioritized based on secondary factorsby the HUD module 115. As shown in FIG. 4c , in this example, the AppleiPhone® may be prioritized over the Samsung Galaxy based on eitherbetter promotional deal exhibited by the Apple iPhone®, which in thisexample costs $100 less or the determining factor in the prioritizationmay be due to the user's 301 previously viewing the Apple iPad® in FIG.4b , which is also an Apple® product (i.e. based on viewing habits/brandinterest). In some embodiments, the prioritization resulting in thesecondary overlay 409 a to be prioritized over secondary overlay 409 bmay have been a combination of multiple factors, including the costsavings of the Apple® promotion in combination with previously viewingother Apple® products in the past.

In some embodiments of the augmented display system 103, the ARprioritization module 105 may further comprise a speech recognitionmodule 117. Embodiments of the speech recognition module 117 may access,control, receive and process voice commands in the form of audio datarecorded by the audio recording system 126. Embodiments of the audiorecording system 126 may comprise one or more microphones receivingaudio data from a user 301. The user 301 may speak to the augmenteddisplay system 103 by speaking into one or more of the microphones. Insome embodiments, the recorded audio input data may be transcribed bythe speech recognition system 117 into text and scanned by the speechrecognition for keywords and/or commands. Upon identifying one or morecommands, the speech recognition system 117 may transmit the commands oraudio inputs to one or more respective modules or the processor 116 forexecution of said commands or inputs by the user 301. For example, theuser 301 may implement voice commands via the audio recording system 126to toggle the primary focus 401 or the prioritization of virtualoverlays 407, 409, 411 being displayed by the HUD of the display device110. As the user 301 inputs voice commands, the voice commands arerecognized by the speech recognition module 126 as a command to togglethe prioritization or primary focus 401. The speech recognition module117 may send a system call to the HUD module 115 directing the HUDmodule 115 to toggle the virtual overlays 407, 409, 411 and/or theprimary focus 401 accordingly.

Referring to the drawings, FIG. 2a to FIG. 2d depict an embodiment of anaugmented display system 103 which may be worn the user 301. As shown inthe figures, the exemplary embodiment of the augmented display system103 may be a pair of glasses comprising a frame 203, a pair of arms 205each comprising a hinge and a pair of lenses 207. The frame 203, arms205 and lenses 207 may be constructed out of any material known by aperson skilled in the art of glasses construction. For example, theunderlying components of the glasses of the augmented display system 103may be constructed out of various plastics, resins, rubbers, metals ormetal alloys, etc. While the exemplary embodiment of the augmenteddisplay system 103 may be depicted as glasses, this should in no way belimiting to the appearance that the augmented display system 103 maytake. Glasses are merely one example and the augmented display system103 may take other forms that comprise computer system capable ofoverlaying images or video data projected by the computer system onto adisplay device 110 having a HUD and GUI overlaid or juxtaposed withinthe vicinity of one or more objects 403, 405, 406 or digitalrepresentation 503, 505, 506, as an object is being viewed in real timeby a visual recording system 124. For example, a mobile device or tabletcomputer shown in FIG. 3a-3b and FIG. 5 acting as an augmented displaysystem 103 having a display device 110, visual recording system 124 andaudio recording system 126 as shown.

The embodiment of the glasses used as an augmented display system 103may include electrical and computing components integrated into theglasses themselves. For example, a projection device may be installedwithin the housing 201 attached to the frame 203 or arms 205 of theaugmented display system 103 as shown in FIG. 2a-2d . Within theinterior of the housing 201, the computer system components integratedtherein may include any of the components described above for theaugmented display system 103 and may integrate components of the genericcomputer system 900 of FIG. 9 discussed in detail below, including aprocessor 991, memory devices 994, 995 an input device 992 and an outputdevice 993. Additional specialized hardware and software components thatmay be integrated into glasses embodiment may include the profile module107, visual recognition module 108, display controller 109, overlaymodule 113, HUD module 115, speech recognition module 117, analyticsmodule 119, display device 110, visual recording system 124 and audiorecording system 126.

Method for Prioritizing a Stacking Order of a Z-Index

The drawing of FIG. 8 represents an embodiment of an algorithm 800 thatmay be implemented for prioritizing a stacking order of a z-index ofvirtual overlays in an augmented display system 103, in accordance withthe systems described in FIGS. 1-7 using one or more computer systemsdefined generically in FIG. 9 below, and more specifically by thespecific embodiments depicted in FIGS. 1-7. A person skilled in the artshould recognize that the steps of the method described in FIG. 8 maynot require all of the steps disclosed herein to be performed, nor doesthe algorithm 800 of FIG. 8 necessarily require that all the steps beperformed in the particular order presented. Variations of the methodsteps presented in FIG. 8 may be performed in a different order thanpresented.

The algorithm 800 described in FIG. 8 may describe a process forprioritizing a stacking order of a z-index using an augmented displaysystem 103. The algorithm 800 may initiate in step 801 by detecting andidentifying one or more objects viewed through the augmented displaysystem 103. The detection and identification of each object may be madeby the visual recognition module 108, which may be receiving images orvideo data streamed by one or more cameras of the visual recognitionsystem 124. As the visual recognition module 108 receives the images andvideo data, the visual recognition module 108 may implement one or moreobject recognition techniques to detect and identify the objectscurrently within the frame of the image or video. Once an object hasbeen detected by the visual recognition system 108, the algorithm 800may proceed to step 805 and determine an objects' importance orrelevance in the stacking order of the z-index.

In step 805, the visual recognition module 108 may begin the steps fordetermining a level of importance of each object detected in step 801.In step 805, a determination may be made by the visual recognitionmodule 108 whether or not the detected object is a primary focus 401 ofthe user 301 using one or more techniques. For example, by automaticallytracking the user's 301 gaze, the focus of one or more cameras in thevisual recording system 124 or any manual object selections that a usermay have chosen to focus on. If the detected object is an object ofprimary focus 401, the algorithm 800 may proceed to step 807, whereinthe visual recognition module 108 may assign the object an identifier asa primary object 403. The algorithm 800 may proceed to step 813 anddetermine whether or not additional objects are detected and/or need anassignment of object type. If additional objects exist within thecurrent viewing distance of the augmented display system 103, thealgorithm may proceed back to step 801 and continue to identify thedetected objects, otherwise the algorithm may proceed to step 815.

In step 805, the determination made by the visual recognition module 108may be that the object detected and identified in step 801 is not theprimary focus 401 of the user 301. The algorithm 800 may proceed to step803 and make a second determination regarding whether or not thedetected object in step 801 is similar in some way to an object ofprimary focus 401. For example, is the detected object a similar type,classification, category or functionality to the object of primaryfocus. If the object is similar in some manner, the algorithm 800 mayproceed to step 811, wherein the object may be assigned a designation asa secondary object 406 by the visual recognition module 108. Likewise,if the object is found not to have a similar type, classification,categorization, etc. in step 803, the algorithm may proceed to step 809,wherein the visual recognition module 108 may assign the object adesignation as a tertiary object 405. Once the objects have beendesignated with a classification in step 809 or 811, the algorithm mayproceed to step 813 as described above.

In step 815, the system 100 may perform the step querying one or moredata sources, such as an overlay data source 131, network repository 133or local data storage device 118 for one or more available virtualoverlays 407, 409, 411 corresponding to each object designated in step807, 809 or 811 as described above. This querying step may be performedby the overlay module 113 in some embodiments. Subsequently, in responseto the query results, the algorithm 800 may continue to retrieve each ofthe virtual overlays 407, 409, 411 from the one or more data sources.The virtual overlays 407, 409, 411 may be loaded into the memory device114 of the augmented display system 103 or into an onboard memory deviceof the AR prioritization module 105 and/or a memory device integratedinto one or more sub-modules thereof.

In step 819 of the algorithm 800, the HUD module 115 may begingenerating and building a HUD comprising each virtual overlay 407, 409,411 retrieved from the one or more data sources in step 819. In step421, the HUD module 115 may prioritize the stacking order of the z-indexfor the HUD being generated by the system in step 819. The HUD module115 may prioritize the stacking order in manner such that the z-indexassigns primary overlays 407 as the highest priority followed by thesecondary overlays 409 and the tertiary overlays 411 are assigned thelowest priority.

Embodiments of the algorithm 800 may cross check each of the primaryobjects 403, secondary objects 406 or tertiary objects 405 to determineif there are multiple co-equal objects within one or more objectclassifications. In step 823 of the algorithm 800, the HUD module 115may determine whether any co-equally classified objects or overlaysthereof overlap. If there is an overlap between objects of a co-equalclassification, the algorithm 800 may proceed to step 825, wherein theHUD module 115 applies one or more secondary factors to create a z-indexhierarchy within each co-equal object level. Once all co-equal objectshave been prioritized within the stacking order of the z-index, thealgorithm may proceed to step 825. Likewise, if in step 823, there isnot an overlay of co-equal object overlays, the algorithm 800 may skipstep 825 and proceed directly to step 827. In step 827, the HUDgenerated in step 819 comprising a prioritized set of virtual overlays407, 409, 411 in steps 821-825 and the HUD may be displayed on thedisplay device 110 of the augmented display system 103.

Computer System

Referring to the drawings, FIG. 9 illustrates a block diagram of acomputer system 900 that may be included in the systems of FIGS. 1-7 andfor implementing methods for prioritizing a stacking order of a z-indexof virtual overlays of augmented reality in accordance with theembodiments described in the present disclosure. The computer system 900may generally comprise a processor 991, otherwise referred to as acentral processing unit (CPU), an input device 992 coupled to theprocessor 991, an output device 993 coupled to the processor 991, andmemory devices 994 and 995 each coupled to the processor 991. The inputdevice 992, output device 993 and memory devices 994, 995 may each becoupled to the processor 991 via a bus. Processor 991 may performcomputations and control the functions of computer 900, includingexecuting instructions included in the computer code 997 for tools andprograms for prioritizing a stacking order of a z-index of virtualoverlays of augmented reality, in the manner prescribed by theembodiments of the disclosure using the systems of FIGS. 1-7, whereinthe instructions of the computer code 997 may be executed by processor991 via memory device 995. The computer code 997 may include software orprogram instructions that may implement one or more algorithms forimplementing the methods for prioritizing a stacking order of a z-indexof virtual overlays of augmented reality, as described in detail above.The processor 991 executes the computer code 997. Processor 991 mayinclude a single processing unit, or may be distributed across one ormore processing units in one or more locations (e.g., on a client andserver).

The memory device 994 may include input data 996. The input data 996includes any inputs required by the computer code 997, 998. The outputdevice 993 displays output from the computer code 997, 998. Either orboth memory devices 994 and 995 may be used as a computer usable storagemedium (or program storage device) having a computer readable programembodied therein and/or having other data stored therein, wherein thecomputer readable program comprises the computer code 997, 998.Generally, a computer program product (or, alternatively, an article ofmanufacture) of the computer system 900 may comprise said computerusable storage medium (or said program storage device).

Memory devices 994, 995 include any known computer readable storagemedium, including those described in detail below. In one embodiment,cache memory elements of memory devices 994, 995 may provide temporarystorage of at least some program code (e.g., computer code 997, 998) inorder to reduce the number of times code must be retrieved from bulkstorage while instructions of the computer code 997, 998 are executed.Moreover, similar to processor 991, memory devices 994, 995 may resideat a single physical location, including one or more types of datastorage, or be distributed across a plurality of physical systems invarious forms. Memory devices 994, 995 can include data distributedacross, for example, a local area network (LAN) or a wide area network(WAN). Further, memory devices 994, 995 may include an operating system(not shown) and may include other systems not shown in the figures.

In some embodiments, rather than being stored and accessed from a harddrive, optical disc or other writeable, rewriteable, or removablehardware memory device 994, 995, stored computer program code 998 (e.g.,including algorithms) may be stored on a static, non-removable,read-only storage medium such as a Read-Only Memory (ROM) device 999, ormay be accessed by processor 991 directly from such a static,non-removable, read-only medium 999. Similarly, in some embodiments,stored computer program code 997 may be stored as computer-readablefirmware 999, or may be accessed by processor 991 directly from suchfirmware 999, rather than from a more dynamic or removable hardwaredata-storage device 995, such as a hard drive or optical disc.

In some embodiments, the computer system 900 may further be coupled toan input/output (I/O) interface 112 and a computer data storage unit(for example a data store, data mart or repository). An I/O interface112 may include any system for exchanging information to or from aninput device 992 or output device 993. The input device 992 may be,inter alia, a keyboard, joystick, trackball, touchpad, mouse, sensors,beacons, RFID tags, audio recording system 126 including one or moremicrophones, biometric input device, visual recording system 124comprising a camera system, a timer, etc. The output device 993 may be,inter alia, a printer, a plotter, a display device 110 (such as acomputer screen, monitor or device capable of projecting an imagethereon), a magnetic tape, a removable hard disk, a floppy disk, etc.The memory devices 994 and 995 may be, inter alia, a hard disk, a floppydisk, a magnetic tape, an optical storage such as a compact disc (CD) ora digital video disc (DVD), a dynamic random access memory (DRAM), aread-only memory (ROM), etc. The bus may provide a communication linkbetween each of the components in computer 900, and may include any typeof transmission link, including electrical, optical, wireless, etc.

The I/O interface 112 may allow the computer system 900 to storeinformation (e.g., data or program instructions such as program code997, 998) on and retrieve the information from a computer data storageunit (not shown in FIG. 9). Computer data storage units include anyknown computer-readable storage medium, which is described below. In oneembodiment, computer data storage unit may be a non-volatile datastorage device, such as a magnetic disk drive (i.e., hard disk drive) oran optical disc drive (e.g., a CD-ROM drive which receives a CD-ROMdisk).

As will be appreciated by one skilled in the art, in a first embodiment,the present invention may be a method; in a second embodiment, thepresent invention may be a system; and in a third embodiment, thepresent invention may be a computer program product. Any of thecomponents of the embodiments of the present invention can be deployed,managed, serviced, etc. by a service provider able to deploy orintegrate computing infrastructure with respect to prioritizing astacking order of a z-index of virtual overlays of augmented reality.Thus, an embodiment of the present invention discloses a process forsupporting computer infrastructure, where the process includes providingat least one support service for at least one of integrating, hosting,maintaining and deploying computer-readable code (e.g., program code997, 998) in a computer system (e.g., computer 800) including one ormore processor(s) 991, wherein the processor(s) carry out instructionscontained in the computer code 997 causing the computer system toprioritize a z-index of augmented reality overlays. Another embodimentdiscloses a process for supporting computer infrastructure, where theprocess includes integrating computer-readable program code into acomputer system including a processor.

The step of integrating includes storing the program code in acomputer-readable storage device of the computer system through use ofthe processor. The program code, upon being executed by the processor,implements a method for prioritizing a stacking order of a z-index ofvirtual overlays of augmented reality. Thus the present inventiondiscloses a process for supporting, deploying and/or integratingcomputer infrastructure, integrating, hosting, maintaining, anddeploying computer-readable code into the computer system 900, whereinthe code in combination with the computer system 900 is capable ofperforming a method of prioritizing a stacking order of a z-index ofvirtual overlays of augmented reality.

A computer program product of the present invention comprises one ormore computer readable hardware storage devices having computer readableprogram code stored therein, said program code containing instructionsexecutable by one or more processors of a computer system to implementthe methods of the present invention.

A computer program product of the present invention comprises one ormore computer readable hardware storage devices having computer readableprogram code stored therein, said program code containing instructionsexecutable by one or more processors of a computer system to implementthe methods of the present invention.

A computer system of the present invention comprises one or moreprocessors, one or more memories, and one or more computer readablehardware storage devices, said one or more hardware storage devicescontaining program code executable by the one or more processors via theone or more memories to implement the methods of the present invention.

The present invention may be a system, a method, and/or a computerprogram product at any possible technical detail level of integration.The computer program product may include a computer readable storagemedium (or media) having computer readable program instructions thereonfor causing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, configuration data for integrated circuitry, oreither source code or object code written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Smalltalk, C++, or the like, and procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The computer readable program instructions may executeentirely on the user 310's computer, partly on the user 310's computer,as a stand-alone software package, partly on the user 310's computer andpartly on a remote computer or entirely on the remote computer orserver. In the latter scenario, the remote computer may be connected tothe user 310's computer through any type of network, including a localarea network (LAN) or a wide area network (WAN), or the connection maybe made to an external computer (for example, through the Internet usingan Internet Service Provider). In some embodiments, electronic circuitryincluding, for example, programmable logic circuitry, field-programmablegate arrays (FPGA), or programmable logic arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the blocks may occur out of theorder noted in the Figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

What is claimed:
 1. A method for prioritizing a stacking order of az-index of virtual overlays in an augmented display system comprisingthe steps of: determining, by a processor, a primary focus of a useroperating an augmented display system; detecting, by a processor, one ormore objects viewed through the augmented display system; designating,by the processor, each of the one or more objects as a primary objectand a secondary object as a function of the primary focus of the user;assigning, by the processor, the stacking order for each of the virtualoverlays corresponding to each primary object and secondary object inthe z-index of a heads-up display (HUD) of the augmented display systemgenerated by the processor, wherein a stacking order of the virtualoverlays of the primary object is higher than a stacking order of thevirtual overlays of each secondary object; and displaying, by theprocessor, each of the virtual overlays in the z-index of the HUD inaccordance with the stacking order assigned, and juxtaposing each of thevirtual overlays near each corresponding primary object or secondaryobject.
 2. The method of claim 1, wherein the stacking order of two ormore virtual overlays of secondary objects are further assigned as afunction of secondary factors selected from the group consisting ofrelevance to a user, social relationship, purchase history, environment,cognitive state and value.
 3. The method of claim 1, wherein thestacking order of two or more virtual overlays are assigned as afunction of a value of a promotion, wherein a promotion having a highestpromotional value is stacked highest in the stacking order and thestacking order dynamically adjusts based on the value of the promotionas the value of each promotion changes in real time.
 4. The method ofclaim 1, wherein the stacking order of two or more virtual overlays areassigned as a function of brand preference to the user, wherein morepreferable brands are stacked higher in the stacking order than lesspreferable brands.
 5. The method of claim 1, wherein the stacking orderof two or more virtual overlays are assigned as a function of asurrounding environment, wherein a primary object or secondary objectthat is more relevant to the surrounding environment is stacked higherin the stacking order.
 6. The method of claim 1, wherein the stackingorder of two or more virtual overlays are assigned as a function of acognitive state of the user and virtual overlays corresponding to aprimary object or secondary object more relevant to the user's cognitivestate is stacked higher in the stacking order.
 7. The method of claim 1,further comprises the steps of: recording by a camera of a visualrecording system video or images of the primary object or the secondaryobject to the display device; and the step of displaying furthercomprises displaying the recorded video or images of the primary objectand secondary object as virtual representations on the display device aspart of the HUD.
 8. The method of claim 1, further comprising providingat least one support service for at least one of creating, integrating,hosting, maintaining, and deploying computer-readable program code in acomputer system, where the computer-readable program code in combinationwith the computer system is configured to implement the steps ofdetermining, detecting, designating, retrieving, assigning, anddisplaying.
 9. A computer system, comprising: a processor; a memorydevice coupled to the processor; a visual recording system comprising acamera, coupled to the processor; a display device, coupled to theprocessor; and a computer readable storage device coupled to theprocessor, wherein the storage device contains program code executableby the processor via the memory device to implement a method forprioritizing a stacking order of a z-index of virtual overlayscomprising the steps of: determining, by the processor, a primary focusof a user operating an augmented display system; detecting, by aprocessor, one or more objects viewed through the augmented displaysystem; designating, by the processor, each of the one or more objectsas a primary object and a secondary object as a function of the primaryfocus of the user; assigning, by the processor, the stacking order foreach of the virtual overlays corresponding to each primary object andsecondary object in the z-index of a heads-up display (HUD) of theaugmented display system generated by the processor, wherein a stackingorder of the virtual overlays of the primary object is higher than astacking order of the virtual overlays of each secondary object; anddisplaying, by the processor, each of the virtual overlays in thez-index of the HUD in accordance with the stacking order assigned, andjuxtaposing each of the virtual overlays near each corresponding primaryobject or secondary object.
 10. The computer system of claim 9, whereinthe stacking order of two or more virtual overlays of secondary objectsare further assigned as a function of secondary factors selected fromthe group consisting of relevance to a user, social relationship,purchase history, environment, cognitive state and value.
 11. Thecomputer system of claim 9, wherein the stacking order of two or morevirtual overlays are assigned as a function of a value of a promotion,wherein a promotion having a highest promotional value is stackedhighest in the stacking order and the stacking order dynamically adjustsbased on the value of the promotion as the value of each promotionchanges in real time.
 12. The computer system of claim 9, wherein thestacking order of two or more virtual overlays are assigned as afunction of brand preference to the user, wherein more preferable brandsare stacked higher in the stacking order than less preferable brands.13. The computer system of claim 9, wherein the stacking order of two ormore virtual overlays are assigned as a function of a surroundingenvironment, wherein a primary object or secondary object that is morerelevant to the surrounding environment is stacked higher in thestacking order.
 14. The computer system of claim 9, further comprises:recording by the camera of the visual recording system video or imagesof the primary object or the secondary object to the display device; anddisplaying further comprises displaying the recorded video or images ofthe primary object and secondary object as virtual representations onthe display device as part of the HUD.
 15. The computer system of claim9, wherein the primary objects and secondary objects are re-designatedand the stacking order of the virtual overlays are dynamicallyreassigned as a function of a change in the primary focus of a user. 16.A computer program product comprising: one or more computer readablehardware storage devices having computer readable program code storedtherein, said program code containing instructions executable by the oneor more central processing units (CPU) of a computer system implementinga method for comparing products using augmented reality comprising thesteps of: determining, by the CPU, a primary focus of a user operatingan augmented display system; detecting, by a CPU, one or more objectsviewed through the augmented display system; designating, by theprocessor, each of the one or more objects as a primary object and asecondary object as a function of the primary focus of the user;assigning, by the processor, the stacking order for each of the virtualoverlays corresponding to each primary object and secondary object inthe z-index of a heads-up display (HUD) of the augmented display systemgenerated by the processor, wherein a stacking order of the virtualoverlays of the primary object is higher than a stacking order of thevirtual overlays of each secondary object; and displaying, by theprocessor, each of the virtual overlays in the z-index of the HUD inaccordance with the stacking order assigned, and juxtaposing each of thevirtual overlays near each corresponding primary object or secondaryobject.
 17. The computer program product of claim 16, furthercomprising: recording by a camera of a visual recording system video orimages of the primary object or the secondary object to the displaydevice; and the step of displaying further comprises displaying therecorded video or images of the primary object and secondary object asvirtual representations on the display device as part of the HUD. 18.The computer program product of claim 16, wherein the stacking order oftwo or more virtual overlays are assigned as a function of a value of apromotion, wherein a promotion having a highest promotional value isstacked highest in the stacking order and the stacking order dynamicallyadjusts based on the value of the promotion as the value of eachpromotion changes in real time.
 19. The computer program product ofclaim 16, wherein the stacking order of two or more virtual overlays areassigned as a function of brand preference to the user, wherein morepreferable brands are stacked higher in the stacking order than lesspreferable brands.
 20. The computer program product of claim 16, whereinthe stacking order of two or more virtual overlays are assigned as afunction of a surrounding environment, wherein a primary object orsecondary object that is more relevant to the surrounding environment isstacked higher in the stacking order.